Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. The primary difficulty arises due to insufficient exploration, resulting in an agent being unable to learn robust value functions. Intrinsically motivated agents can explore new behavior for its own sake rather than to directly solve problems. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical value functions, operating at different temporal scales, with intrinsically motivated deep reinforcement learning. A top-level value function learns a policy over intrinsic goals, and a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse, delayed feedback: (1) a complex discrete stochastic decision process, and (2) the classic ATARI game 'Montezuma's Revenge'.

We consider the task of text generation in language models with constraints specified in natural language. To this end, we first create a challenging benchmark Cognac that provides as input to the model a topic with example text, along with a constraint on text to be avoided. Unlike prior work, our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata, which allows for straightforward evaluation while striking a balance between broad attribute-level and narrow lexical-level controls. We find that even state-of-the-art language models like GPT-3 fail often on this task, and propose a solution to leverage a language model's own internal knowledge to guide generation. Our method, called CognacGen, first queries the language model to generate guidance terms for a specified topic or constraint, and uses the guidance to modify the model's token generation probabilities. We propose three forms of guidance (binary verifier, top-k tokens, textual example), and employ prefix-tuning approaches to distill the guidance to tackle diverse natural language constraints. Through extensive empirical evaluations, we demonstrate that CognacGen can successfully generalize to unseen instructions and outperform competitive baselines in generating constraint conforming text.

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Fine-tuning pre-trained language models (PLMs) achieves impressive performance on a range of downstream tasks, and their sizes have consequently been getting bigger. Since a different copy of the model is required for each task, this paradigm is infeasible for storage-constrained edge devices like mobile phones. In this paper, we propose SPARTAN, a parameter efficient (PE) and computationally fast architecture for edge devices that adds hierarchically organized sparse memory after each Transformer layer. SPARTAN freezes the PLM parameters and fine-tunes only its memory, thus significantly reducing storage costs by re-using the PLM backbone for different tasks. SPARTAN contains two levels of memory, with only a sparse subset of parents being chosen in the first level for each input, and children cells corresponding to those parents being used to compute an output representation. This sparsity combined with other architecture optimizations improves SPARTAN's throughput by over 90% during inference on a Raspberry Pi 4 when compared to PE baselines (adapters) while also outperforming the latter by 0.1 points on the GLUE benchmark. Further, it can be trained 34% faster in a few-shot setting, while performing within 0.9 points of adapters. Qualitative analysis shows that different parent cells in SPARTAN specialize in different topics, thus dividing responsibility efficiently.

While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines, as well as improved human interpretability and trustworthiness over methods without reasoning or acting components. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generates human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples. Project site with code: https://react-lm.github.io

为了识别动态网络中嵌入的系统（模块），必须制定一个多输入估计问题，该问题需要测量某些节点并将其作为预测输入。但是，由于传感器选择和放置问题，在许多实际情况下，其中一些节点可能无法测量。这可能会导致目标模块的偏差估计。此外，与多输入结构相关的识别问题可能需要确定实验者不特别感兴趣的大量参数，并且在大型网络中的计算复杂性增加。在本文中，我们通过使用数据增强策略来解决这些问题，该策略使我们能够重建缺失的节点测量并提高估计目标模块的准确性。为此，我们使用基于正规化的基于内核的方法和近似推理方法开发了系统识别方法。为感兴趣的模块保留一个参数模型，我们将其他模块作为高斯过程（GP）建模，并用所谓的稳定样条核给出的内核。经验贝叶斯（EB）方法用于估计目标模块的参数。相关的优化问题是使用预期最大化（EM）方法来解决的，在该方法中，我们采用马尔可夫链蒙特卡洛（MCMC）技术来重建未知的缺失节点信息和网络动力学。动态网络示例上的数值模拟说明了开发方法的电势。

骨肉瘤是最常见的原发性骨癌，其标准治疗包括术前化疗，然后切除。化学疗法反应用于预测患者的预后和进一步治疗。坏死在切除标本上的组织学幻灯片通常评估了坏死比定义为坏死肿瘤与总体肿瘤之比。已知坏死比> = 90％的患者的预后更好。多个载玻片对坏死比的手动微观综述是半定量性的，并且可能具有观察者间和观察者间的变异性。我们提出了一种基于目标和可再现的深度学习方法，以估计坏死比，并从扫描的苏木精和曙红全幻灯片图像预测结果。我们以3134个WSI的速度收集了103例骨肉瘤病例，以训练我们的深度学习模型，验证坏死比评估并评估结果预测。我们训练了深层多磁化网络，以分割多个组织亚型，包括生存的肿瘤和像素级中的坏死肿瘤，并计算来自多个WSI的病例级坏死比。我们显示了通过分割模型估算的坏死比，高度与由专家手动评估的病理报告中的坏死比高度相关，其中IV级的平均绝对差异（100％），III（> = 90％）和II（> = 50％和<50％和< 90％）坏死反应分别为4.4％，4.5％和17.8％。我们成功地对患者进行了分层，以预测P = 10^-6的总生存率，而P = 0.012的无进展生存率。我们没有可变性的可重现方法使我们能够调整截止阈值，特别是用于模型和数据集的截止阈值，为OS的80％，PFS为60％。我们的研究表明，深度学习可以支持病理学家作为一种客观的工具，可以分析组织学中骨肉瘤，以评估治疗反应并预测患者结果。

可靠性的关键问题是电路设计师的巨大关注之一。驱动力是晶体管老化，取决于操作电压和工作负载。在设计时，很难估算在终生期间保持衰老效果的近距离护罩。这是因为铸造厂不共享其基于物理的校准模型，该模型由高度机密的技术和材料参数组成。但是，对降解的不受监控但必要的高估相当于绩效下降，这是可以预防的。此外，这些基于物理学的模型在计算方面非常复杂。在设计时间为数百万个单个晶体管建模的成本显然是过高的。我们提出了经过培训的机器学习模型的革命前景，以复制基于物理的模型，以免披露机密参数。出于设计优化的目的，电路设计人员可以完全访问这种有效的解决方法。我们证明了模型通过对一个电路的数据进行训练并将其成功应用于基准电路的能力。平均相对误差高达1.7％，速度高达20倍。电路设计师有史以来首次可以易于使用高精度老化模型，这对于有效的设计至关重要。这项工作是跨越铸造厂和电路设计师之间宽阔鸿沟的方向的一个有希望的步骤。

在交互式环境中，现有的基础语言基准要么缺乏现实世界的语言元素，要么由于人类参与数据收集或反馈信号而难以扩展。为了弥合这一差距，我们开发了网络商店 - 一个模拟的电子商务网站环境，拥有11.18亿美元的现实世界中的产品和12,087美元的人群文本说明。给定指定产品需求的文本指令，代理需要导航多种类型的网页并发布各种操作以查找，自定义和购买项目。 WebShop为语言基础提供了一些挑战，包括了解构图说明，查询（重新）表述，理解和对网页中的嘈杂文本进行操作以及执行战略探索。我们为这项任务收集了超过1,600美元的人类示范，并使用强化学习，模仿学习以及预训练的图像和语言模型来训练和评估各种代理商。我们的最佳模型达到了任务成功率$ 29 \％$，它优于基于规则的启发式方法（$ 9.6 \％$），但远低于人类专家绩效（$ 59 \％$）。我们还分析了代理和人类轨迹，并消融各种模型组件，以提供有关具有更强语言理解和决策能力的未来代理人的见解。最后，我们表明，在Amazon.com上进行评估时，在网络商店进行培训的代理商展示了非平凡的SIM转移转移，这表明网络商店在开发可以在野外运行的实用基于网络的代理商中的潜在价值。

强大而广义的工具操作需要了解不同工具的属性和提供的功能。我们研究有关工具的语言信息（例如，其几何形状，常用用途）是否可以帮助控制策略更快地适应给定任务的新工具。我们获得了自然语言中各种工具的各种描述，并使用预训练的语言模型来生成其功能表示。然后，我们执行语言条件的元学习，以学习可以有效地适应新工具的政策。我们的结果表明，将语言信息和元学习结合起来可以显着加速工具在几个操纵任务中的学习，包括推动，举重，清扫和锤击。